I have noticed that the reason for the difference in results may be due to --full_bf16 or --full_bf16.
It would be better to add is_full argument to sdxl_train_util.load_target_model and convert the unet, text_encoder1, text_encoder2 in it.

Related to #628

This seems like a good feature. I found myself wanting it just last week.

Thank you for this. I think the difference in results may be acceptable.

In addition, I have another issue in generating samples during training (I got strange sepia images), so I cannot confirm the difference yet.

I found that if --sample_every_n_epochs is set, the script generates the images two times every N epochs after merging. I think this is because the following lines are in the training loop (for epoch ...).

        # For --sample_at_first
        self.sample_images(accelerator, args, epoch, global_step, accelerator.device, vae, tokenizer, text_encoder, unet)

To solve this, I will move these two lines just before the loop.